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Abstract 

With the anticipated explosion of small unmanned aerial vehicles, it is highly likely that operators will be controlling fleets of 
autonomous vehicles. To fulfill the promise of autonomy, vehicle operators will not be concerned with manual control of the 
vehicle; instead, they will deal with the overall mission. Furthermore, the one operator to many vehicles is becoming a constant 
meme with various industries including package delivery, search and rescue, and utility companies. In order for an operator to 
concurrently control several vehicles, his station must look and behave very differently than the current ground control station 
instantiations. Furthermore, the vehicle will have to be much more autonomous, especially during non-normal operations, in 
order to accommodate the knowledge deficit or the information overload of the operator in charge of several vehicles. The 
expected usage increase of small drones requires presenting the operational information generated by a fleet of heterogeneous 
autonomous agents to an operator. NASA Langley Research Center’s Autonomy Incubator has brought together researchers in 
various disciplines including controls, trajectory planning, systems engineering, and human factors to develop an integrated 
system to study autonomy issues. The initial human factors effort is focusing on mission displays that would give an operator the 
overall status of all autonomous agents involved in the current mission. This paper will discuss the specifics of the mission 
displays for operators controlling several vehicles. 
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1. Introduction 

As technology has evolved and advanced, lower-level control actions, initially performed by the user, have been 
taken over by machine automation. Thus, the user tasks have traditionally progressed from single- to multi-loop 
control structures (comprised of inner-loop and outer-loop controllers) to eventually adding on-the-loop supervisory 
control. An example of this phenomenon is driving a car. The first cars required the driver to manually crank the 
engine to start with direct steering linkages to the wheels. Cars rapidly progressed to automatic transmission where 
the driver no longer had to shift and had power steering. Next came antilock braking systems eliminating the need of 
the driver pumping the brakes to increase breaking and steering power, and cruise control where the driver no longer 
had to manually maintain a constant speed. Some of the more modem cars now even parallel park for drivers and 
have cmise control that decreases speed automatically when approaching another vehicle. All these innovations, 
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which undoubtedly increased safety [1], have slowly taken the driver farther out of the loop. The ultimate endpoint 
is the Google self-driving car which does not even require a driver to be actively involved in the driving task [2]. 

Similar progressions can be seen in just about every modern day convenience from radios, where a user just 
needs to press a button to tune it, to programmable coffee makers that even grind coffee beans. This evolution is also 
manifested to moving in three-dimensional space. Airplanes have very quickly traversed from direct manual control 
- when the propeller was cranked by hand to start the engine and with direct cable linkages from the control stick to 
the control surfaces - to inner-loop controllers, which stabilized of the vehicle dynamics, to outer-loop control 
where modem day airline transport pilots program the flight management computer on fly-by-wire aircraft. This 
automation, while increasing the safety of air travel immensely, has also generated a slew of new problems 
involving situation awareness and workload of pilots. With the added variables of miniaturization and increased 
automation, now users who are not even pilots can easily fly a small drone [3, 4]. This capability has opened up a 
range of business opportunities [5] - such as crop monitoring, photography and filming, package delivery and 
pipeline inspection, just to name a few [6-1 1] - in addition to the possible increases in public safety in areas such as 
search and rescue (SAR) and fire monitoring [12]. However, in order to make most of these endeavors feasible, a 
single operator will need to be in charge of several vehicles at once. 

This paper does not address the regulatory or liability concerns of autonomous vehicles. In fact, this work 
proposes solutions anathema to current proclamations by the Federal Aviation Administration (FAA). This includes 
the FAA’s reticence towards autonomous systems [13], flying small unmanned aerial vehicles (sUAV) (<55 pounds) 
beyond visual range, non-certificated pilots operating sUAVs, and an operator flying more than one vehicle at a 
time [14, 15]. While these issues must ultimately be addressed, some of the solutions presented in this paper include 
performance bounds that could be eventually used for FAA certification processes. 

This paper will detail the considerations of operators controlling multiple vehicles at a time who may not even 
fully understand the dynamics of the sUAVs under his control. Possible solutions will then be discussed. 


Nomenclature 

FAA Federal Aviation Administration 
SAR search and rescue 

sUAV small unmanned aerial vehicle 

sUxV small unmanned vehicle 

UAV unmanned aerial vehicle 

UxV unmanned vehicle 


2. Problem setting 

For unmanned vehicles (UxV) and possibly for personnel air vehicles, the operator may be partially or 
completely unaware of the dynamics of the UAV and its environment. For example, a user controlling a UAV may 
not fully understand traditional stick and rudder concepts nor may he have a basic understanding of aerodynamics 
such as how stalls occur. In practice, the operator is moving towards being strictly a high-level concepts mission 
manager. He may understand that certain expanses of pipeline need to be inspected; not how to develop the optimal 
routing for each vehicle or how to maintain the flight path and vehicle attitude. Furthermore, the operator may be in 
charge of several vehicles at once with the same general mission such as the search and rescue of lost hikers or 
package deliveries within a defined district. For operational safety, the vehicles and operators should have certain 
characteristics and capabilities. 

2.7. Vehicle characteristics and capabilities 

For the operator to be on-the-loop rather than in-the-loop, the vehicle must have time-coordination and path- 
following capabilities as well as an autopilot that provides automatic stabilization and quickly translates high level 
commands to control surface deflections. Most hobbyist sUAVs have automatic stabilization, and some are able to 
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do waypoint navigation [16, 17]; however, with time-coordination and path- following capabilities, a wider range of 
operators will be able to safely fly these vehicles and will make their operation much more economically viable. 

Another basic characteristic the vehicle needs to safely transit is the ability to sense and safely avoid obstacles, 
and alter the vehicle’s trajectory to accommodate these deviations. Various methods for sensing obstacles have been 
proposed [18, 19] but equivalence to the FAA’s “See and Avoid” has yet to be achieved. 

Furthermore, minimal on-board processing must exist in order to provide robustness to the operator-vehicle 
system when non-normal events occur. These events include lost link from the operator to the vehicle, unplanned 
weather events, and failures on board the vehicle itself such as an engine failure or sensor failure. Robust adaptive 
control techniques, such as L\ [20], can provide nominal or close-to-nominal performance in the case of a failure. 

2.2. Operator characteristics and capabilities 

As mentioned above, the operator may not fully understand the dynamics of the vehicles he is managing. This is 
an on-going trend with vehicles that operate in two and three-dimensional space. For example, fewer car drivers 
understand how to correct out of skids without antilock brakes, and pilots may have a difficult time explaining the 
benefits and aerodynamic effects of flaps. This occurs as automation increases; the operator no longer needs to 
understand the inner-loop control aspects of his tasks - he only needs to understand the tasks at his control level. 

With the increasing capability of UxVs to maintain basic worthiness, the vehicle operator is better described as a 
mission manager. He will be skilled in overseeing the mission. For package delivery, he may be akin to the 
dispatcher but he is dispatching vehicles without a pilot or driver. For SAR, he may be an expert in interpreting 
sensor data indicating a child as opposed to an adult rather than flying sUAVs directly. As a mission manager, he 
will want to understand why things are happening. When an event unexpectedly happens, operators today contact 
the pilot or driver to directly ask questions, or they inherently understand the situation because they are on the scene 
(i i.e ., in the vehicle itself). However, if the vehicle does not have an on-board driver or pilot, then the operator will 
have to rely on the limited data from the vehicle itself. 

Lastly, many missions will require a single operator to be in charge of a fleet of agents in order for the economics 
to work for the available mission resources. This capability is beginning to exist [21, 22] and is further influenced by 
the vehicle on-board capabilities. With fleets of vehicles under his control, the operator may be remotely situated 
from the vehicles or the vehicles may be beyond the operator’s visual range for part of the mission. 

3. Approach: high-level mission manager with transparent decision making among autonomous agents 

The following details proposed solutions for operators overseeing missions with one or more small UxVs (sUxV). 

3.1. Operator’s station 

The operator of sUxVs will often be in charge of planning the mission and then executing the mission because of 
the easy transportability of the small vehicles. For example, a police officer may plan and launch several sUAVs 
from the trunk of his police cruiser for a SAR mission without further infrastructure. 

3.1.1. Mission planning 

Because sUxVs are typically easy to operate, mission planning should be at a high level; therefore, the operator 
station should reflect this. For example, an operator tasked with package delivery should just have to specify the 
starting location, delivery location, return location if it is different from the starting location, delivery time, and 
weight of the package. The underlying algorithms should then be able to determine the appropriate vehicle to carry 
the package and define an appropriate trajectory to deliver the package on time. Figure 1 is an example interface. 
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Fig. 1. Example high-level package delivery planning interface for the mission manager. Includes the ability 
to check for preplanned paths (Package Code), input starting (From) and final location (To) with either 
an address, latitude and longitude, or location code, package weight, and package delivery time. 


However, for a SAR mission or for crop monitoring, the operator’s station may necessarily look very different. 
Instead of defining package weight and delivery time, the operator may need to define the area of interest, the flight 
pattern, the number of vehicles at his disposal, and the sensor signature of interest. Once again, the underlying 
algorithms should be able to define an appropriate trajectory with the given resources. 

The trajectory generation algorithm should generate multiple trajectories that considers each vehicle's dynamics 
and operating characteristics, ensures collision-free maneuvers and guarantees the desired inter- vehicle coordination 
for the specific mission. An example algorithm that considers each vehicle’s dynamics and coordinates the vehicles 
in space and time in order to generate each vehicle’s trajectory is detailed in [23-26] (Fig. 2). This methodology 
employs Pythagorian-Hodograph Bezier curves that guarantee performance bounds for the computed solution. 

3.1.2. Mission monitoring 

Once the mission has started, the operator should be able to easily determine whether all is proceeding as planned 
or what the basic states are of each autonomous vehicle. Once again, this information should be at a high abstraction 
level to help with operator workload especially for missions involving several UxVs. A graphical depiction of where 
each vehicle is and where it is going (Fig. 3a) plus its status are necessary (Fig. 3b). Being able to highlight specific 
vehicles in order to obtain more detailed information may also be helpful (Fig. 4). 

Alerts and alert levels should reflect the autonomous nature of the vehicles. For example, alerts levels may 
indicate the vehicle’s ability to complete its mission. See Table 1 for suggested autonomous vehicle alert levels. 

3.1.3. Transparent decision making 

An important characteristic of the operator and autonomous vehicle system is a shared perception among all 
entities and operator trust of the autonomous vehicles [27]; therefore, the operator must understand what each 
vehicle is doing and why it is doing it. Possible methods to show the operator the decision state of the autonomous 
vehicle includes directed graphs and augmented reality [28-30]. 
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Fig. 2. Three-dimensional temporally deconflicted flight trajectories of 7 multirotors. 



Fig. 3. (a) Example display detailing a vehicle’s location, status, and flight path. 
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Fig. 3. (b) Example display detailing each vehicle’s alert status (green=nominal, 
cyan=advisory, amber=caution, red=waming) on a timeline (current time at Fig. 4. Example display detailing vehicle status, 

magenta line). 


3.1.4. Interaction with the operator station 

The mission manager interacting with the operator’s station may take on many forms besides traditional 
computer keyboard and mouse inputs. For example, many tablets allow for touchscreen interaction and may have 
accelerometers imbedded which would allow for some gesture-like direct control such as tilting the screen to switch 
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Table 1. Suggested autonomous vehicle alert levels and alert level meaning. 

Alert Level Alert Level Meaning 

Advisory Vehicle can complete most of its mission although certain parameters may not be adhered to, such as delivery time or 
completion of the full trajectory, but the vehicle will make its final destination 

Caution Vehicle must immediately return to its starting position or go directly to its end point in order to make it to either of 

those destinations 

Warning Vehicle will not be able to return to its starting position or final destination and will land as close as possible to one of 
those two points 

the display to the next vehicle. Other possibilities include using spoken language [31, 32] and gestures [33, 34] to 
control input. A more natural interaction of the operator with his station may involve a combination of methods; for 
example, using a touchscreen or gestures to indicate that the vehicle needs to land at a specific location on a map but 
using speech to indicate that it needs to fly at a certain altitude and arrive at a certain time. The combination of input 
methods may be much more natural to the operator which would likely decrease workload. 

3.2. On-board processing 

In order for the vehicles to safely operate in the environment, the vehicle must possess some on-board processing 
in addition to basic vehicle attitude control. These functions are (a) path following and replanning (which includes 
maintaining coordinated missions), (b) obstacle detection and avoidance, and (c) vehicle state health assessment. 

These abilities must be on-board the vehicle in cases when the vehicle operator does not respond in time or when 
the communication lag between the vehicle and the operator is longer than the time to an event, such as a collision, 
or if communication between the operator and the vehicle is interrupted or impaired. 

3.2.1. Path following and replanning 

Each vehicle should have its planned trajectory available on-board so that in a lost link event, the vehicle is able 
to complete its mission autonomously if other parameters are nominal. In the case of an operator overseeing a fleet 
of vehicles, each individual vehicle must be able to follow its planned path without operator input because of the 
workload on the operator. The ability for each vehicle to replan its trajectory should also be on-board so that if the 
vehicle must divert due to an obstacle, it can quickly calculate and implement the replanned trajectory. 

3.2.2. Obstacle detection and avoidance 

On-board obstacle detection and avoidance is needed in the event of a lost link with the operator’s station or in 
instances when there is not enough time for the vehicle to coordinate with the operator. Therefore, at least one on- 
board sensor is required. Examples are a traditional RGB [red, green, blue] camera, LIDAR [laser imaging detection 
and ranging], and FLIR [forward-looking infrared radar]. If the obstacle is another air vehicle, then TCAS [traffic 
collision avoidance system] or ADS-B [automatic dependent surveillance-broadcast] are other possibilities. 

In any case, once an obstacle is detected, then the vehicle must be able to avoid it and eventually get back on 
path. An example algorithm that guarantees avoidance along with satisfaction of mission constraints and vehicle 
dynamic constraints is presented in [35] (Fig. 5). 

3.3. Vehicle coordination 

Vehicles may need to coordinate with one another to arrive at a destination at the same time, at prespecified 
times, or in a time window so as to meet given temporal separation requirements [36] (Fig. 6). In this case, the 
communications network must ensure adequate communication between the vehicles [37, 38]. 

4. Conclusion 

This paper suggested and discussed operator requirements to safely plan for and monitor missions of several 
autonomous vehicles. The operator characteristics taken into consideration are a user who may not fully understand 



7 



Fig. 5. Example obstacle avoidance algorithm. 



Fig. 6. (a) Five UAVs arrive at the beginning of glide path within pre- (b) Speed profiles for the transition paths of 5 UAVs 

specified arrival windows and separated by approximately 30 sec. hitting a 10 sec arrival window. 


the vehicle’s dynamics and who may be in charge of a fleet of vehicles that could very well be remotely located 
from him or operate beyond visual range. From these operator characteristics, the attributes of his ground control 
station should allow him to easily plan for and monitor the vehicles for each particular mission. Interaction with the 
operator’s station includes user input employing natural language that may combine input modalities to better enable 
him to interact with the system. With regards to situation awareness, transparent decision making should be included 
so that both the operator and the autonomous entities can fully understand what one another is doing and why. With 
the user’s role as more of a mission manager, the vehicles will require on-board processing in order to sense and 
avoid obstacles, to follow and replan trajectories, and to maintain vehicle coordination. With these characteristics, 
the goals of “[e]nsur[ing] operations in complex, contested environment, [d]emonstrat[ing] highly effective human- 
machine teaming, [c]reat[ing] actively coordinated teams of multiple machines, [and e]nsur[ing] safe and effective 
systems in unanticipated [and] dynamic environments” [39] will be obtained. 
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